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ABSTRACT: 



Characteristics of R&D laboratories are reviewed with emphasis 
on Federal laboratories. Dimensions of the scientist-laboratory 
interface and their potential relationships to R&D products are 
examined. The purpose, need, and approaches to evaluation of 
laboratory effectiveness are reviewed with emphasis on advisory 
panels and criteria for evaluation. Recommendations are made 
for peer-rating the utility of Navy personnel research programs 
from the standpoint of consumer and producer. Recommendations 
are also made for a systematic analysis of the internal environ- 
ment of the Navy's personnel research laboratories as a basis 
for management decisions and programs. 
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PREFACE 



This report deals with a research project between the sponsoring 
activity, the Personnel Research Division of the Bureau of Naval Personnel, 
and the Naval Postgraduate School. The topics examined in this report are: 
(1) Current procedures and recommended, improved procedures for evaluating 
the performance and effectiveness of the Navy Personnel Research and Devel- 
opment Laboratory and the Navy Personnel and Training Research Laboratory 
and (2) administrative and managerial practices for encouraging greater 
application of new research techniques and procedures in the aforementioned 
laboratories. Subsequent reports will deal with the further development 
and application of evaluative procedures, effectiveness measures, and organ- 
izational principles for personnel and training research. 

The study was performed under the auspices of Capt. G. F. Britner, 
Division Director, and Mr. A. A. Sjoholm, Technical Director, Personnel 
Research Division. 

The author thanks Dr. E. I. Jones and Mr. E. M. Ramras , technical 
directors of the Navy Personnel and Training Research Laboratory and Navy 
Personnel Research and Development Laboratory, respectively, for reviewing 
the draft of this study and making appropriate suggestions for the final 
product and future direction of the research effort. 
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I. INTRODUCTION 



The two problems analyzed in t-his report are methods for evalu- 
ating the performance effectiveness of the Navy ! s in-house personnel 
research laboratories and means for improving their performance through 
the adoption and application of new techniques and procedures. These 
two problems can be considered together since they are related to the 
more general question, What characterizes an effective, in-house, Federal 
laboratory? They are related to this question because, first, the 
characteristics of an effective laboratory could provide the basis for 
devising criteria to evaluate the effectiveness of a given laboratory. 

And such characteristics, when known, might also provide guidance for 
recommending managerial and administrative practices to enhance the 
effectiveness of a particular laboratory through the use of new and 
innovative procedures. 

Accordingly, the approach taken in this report is to begin with 
the identification of salient features of organizational climates that 
appear to be important to effective performance within research and 
development (R&D) laboratories. Next, the environment or climate of 
Federal, in-house laboratories will be examined. Following this, attempts 
to assess and evaluate the effectiveness of R&D organization will be 
examined with particular reference to in-house laboratories. Finally, 
the specific procedures currently in use to evaluate the effectiveness 
of the Navy’s in-house, personnel research activities will be reviewed, 
and recommendations will be made for improved procedures based on the 
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information developed in the preceding investigation. 

Recommendations for administrative and managerial practices to 
foster the adoption and application of new research techniques and pro 
cedures in the Navy's personnel research laboratories will arise as a 
by-product of the attempt to identify evaluation criteria and methods 
for assessing the effectiveness of laboratories in the manner contem- 
plated. 
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II. CHARACTERISTICS OF RESEARCH AND DEVELOPMENT LABORATORIES 
A. GENERAL TRENDS 

Leaving undefined for the present what constitutes effectiveness 
in R&D activities, it seems appropriate to assume that managerial and 
organizational practices that are conducive to effective operations in 
any type of organization are also applicable to R&D organizations, such 
as laboratories. This assumption is examined in detail by Argyris (1968) 
in its negative aspects. He explains how the detrimental features of a 
typical pyramidal organizational structure can be magnified in the dy- 
namic life of an R&D organization. The thrust of his argument is that 
typical pyramidal organizations, when imposed on an R&D activity, may be 
conducive to the steady deterioration of the activity. The deterioration 
is primarily the result of improper controls imposed by the upper levels 
of the organization that are based on mistrust, antagonism, and defen- 
siveness. As a result, the system becomes increasingly rigid; less 
competent personnel reach upper levels of the organization; accurate in- 
formation ceases to flow through the system; paternalism begins to 
flourish; more time is spent on generating valid, perhaps, but not very 
important information; and nonresearch activities — "selling," budgets, 
hiding bad news from the top — begin to take precedence over research 
activities. The increasing frustration of the scientists in the organi- 
zation is reflected by hesitating to take on responsibility and making 
tangential demands for space, equipment, assistants, travel to professional 
meetings, and the like. Good scientists leave the organization. As the 
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R&D organization that is locked in to this deteriorating process grows 

older, top management becomes disenchanted with it. Increasing . . . 

intervention by the line departments, tighter research 
budgets, more systematic evaluation of research, more 
rigorous (hopefully) quantitative indices for evaluat- 
ing research payoff, more use of charts to control the 
flow of research, closer link with the marketplace and 
a weaker tie with basic research, and finally, an in- 
crease in the use of top-level committees to oversee 
research can be expected. (Argyris, 1968, p. 353.) 

The positive side of this argument is to design the controls, 

organizational structures, and leadership styles so that the participants 

. . may give of themselves rather than give u£ themselves" (Argyris, 

1968, p. 347). The basis for designing such controls rests in openness, 

respect for individuality, trust, concern, and a willingness to permit 

risk-taking and innovation. 

Controls, of course, imply pressure regardless of whether the 
controls are good or bad, positive or negative. To most persons, pressure 
in a job is an indication of undesirable stresses. In fact, the often 
asked question about an individual is his ability to stand up under 
pressure. Thus, a logical next step, following Argyris 1 critical ap- 
praisal of controls in R&D organizations, is an examination of the role 
of pressure. Should pressure be entirely undesirable? If so, how are 
controls to be exercised? Merely through reward systems? The question, 
according to Hall and Lawler (1971) , is not whether pressure is good or 
bad but to determine when pressure is helpful and when it is dysfunc- 
tional. They state that healthy persons have a need to experience in- 
ternal pressure and will create it if the environment fails to provide it. 
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In order to identify healthy pressures and their beneficial uses, 
it is necessary to distinguish between pressure and such terms as conflict, 
stress, tension, anxiety, and strain. To Hall and Lawler, job pressures 
are forces a person experiences which motivate him to behave in partic- 
ular ways on the job, and pressure is the experience of a particular 
force. Strain, conflict, and similar conceptions represent the oppo- 
sition of two or more pressures. With pressure thus defined, their re- 
search identified three kinds of pressure that were mentioned from 
laboratory to laboratory with considerable agreement as to their impor- 
tance to research personnel. These were time, quality, and financial 
responsibility. Time was a felt need to meet schedules and deadlines, 
and it was typically externally imposed. Quality pressure was a concern 
for doing a good technical job and seemed to be generated by the re- 
searchers themselves, along with their colleagues. And finally, financial 
responsibility was felt as a concern for the financial goal-attainment 
of the organization as a whole. 

These pressures were then related by Hall and Lawler (1971) to 
the satisfaction and job involvement of the research worker and to the 
effectiveness of the organization. Quality pressure was related posi- 
tively to both job involvement of the individual professionals and to 
the technical effectiveness of their laboratories . Job involvement was 
also related to laboratory effectiveness. Financial responsibility 
pressure was significantly correlated with both laboratory effectiveness 
and the satisfaction of the researcher's need for autonomy. The impli- 
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cation here was that the more a person helps to support himself and his 
organization financially, the less dependent he is on the organization. 
Time pressure, while not negatively related to any of the criteria, was 
not related in a systematic way to researcher needs or to laboratory 
effectiveness . 

Hall and Lawler (1971) conclude: 

. . . the most effective organizations are those in which 
the professional value of technical excellence and/or the 
organizational value of financial responsibility are in- 
ternalized by the system as a whole. These two alternatives 
indicate two separate types of adaptations — pressure for 
quality reflects organizational adaption to professional 
norms, while financial pressure suggests the professionals 1 
adaption to organizational values. (P. 70.) 

This means, therefore, that in the effective system, 
each member feels "on top of” the organization; he is 
aware of the total system as it relates to its environment 
and knows how his job ties in with the total goals of the 
organization. This self and system awareness represents 
a strong sense of organizational identification for the 
person, and this experience when widely shared is an im- 
portant dimension of organizational health. (P. 71.) 

Pelz and Andrews (1966) , in their work on scientists in organi- 
zations, also conclude that ”. . . the scientists and engineers whom we 
studied did effective work under conditions that were not completely 
comfortable, but contained ’creative tensions’ among forces pulling in 
different directions" (p. 7). They (Pelz & Andrews, 1966; p. 7) 
characterized broad features of the environment of the most productive 
scientists and engineers, as follows: 

‘Effective scientists were self-directed by their own 
ideas, and valued freedom. But at the same time they 
allowed several other people a voice in shaping their 
directions; they interacted vigorously with colleagues. 
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‘Effective scientists did not limit their activities 
either to the world of "application" or to the world 
of "pure science" but maintained an interest in both; 
their work was diversified. 

‘Effective scientists were not fully in agreement with 
their organization in terms of their interests; what 
they personally enjoyed did not necessarily help them 
advance in the structure. 

‘Effective scientists tended to be motivated by the same 
kinds of things as their colleagues. At the same time, 
however, they differed from their colleagues in the 
styles and strategies with which they approached their 
work. 

* In effective older groups, the members interacted vig- 
orously and preferred each other as collaborators, yet 
they held each other at an emotional distance and felt 
free to disagree on technical strategies. 

From the literature reviewed above, it can be said — in general — 
that productive and efficient R&D laboratories use organizational con- 
trols that embody pressures which are not dysfunctional, and their individual 
scientists, while preserving their own independence and interests, are 
flexible and dedicated to both the pursuit of quality in their work and 
to the welfare of their organizations. 



B. TRENDS IN FEDERAL LABORATORIES 



Glass (1970, p. 11) writes that "Probably no class of institution 
has been studied and analyzed, praised and criticized, organized and 
reorganized to the degree that has been the lot of the Defense labora- 
tories." His report (Glass, 1970) provides a comprehensive review of 
the study, analysis, and evolution of Defense in-house laboratories. 
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The status and potential of Federal laboratories are also discussed in 
detail in the report of the hearings in March and April 1968 of the House 
Subcommittee on Science, Research, and Development, frequently referred 
to as the Daddario Committee. (Hereafter it will be referred to as the 
Subcommittee, 1968^) Pelz and Andrews (1966) also devote much of their 
book to scientists in government laboratories. More recently, the Defense 
Blue Ribbon Panel completed its investigation of Defense R&D activities 
with recommendations for major changes and improvements in the system. 
Especially germane to this report is the investigation of the Manpower 
Research Task Force of the Defense Science Board, which is nearing 
completion. Accordingly, the intent here is not to retrace this path of 
study and analysis, praise and criticism. Rather, the purpose of this 
section is to highlight salient characteristics of Federal and especially 
Department of Defense (DoD) laboratories that may have a bearing on the 
questions being examined in this report. 

If professionals in good R&D laboratories identify with their 
organizations as mentioned above, surely those in Federal laboratories 
must do so with respect to the mission and purpose of their laboratories 
and the problems that are selected for investigation. The Committee on 
Federal Laboratories, for example, found that the single most important 
factor in laboratory morale was a sense of purpose on the part of each 
scientist and a sense that the results mattered to someone (Subcommittee, 
1968). Harold B. Finger, Associate Administrator for Organization and 
Management, National Aeronautics and Space Administration (NASA), testi- 
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fying before the Subcommittee, stated that the NASA experience confirmed 
the value and importance of the purpose of an organization. He noted 
that it would be very hard for a research organization to succeed if its 
basic purposes and motivation seemed unimportant; on the other hand, a 

i 

research organization which has an important and significant role to ful- 
fill has a very good start in the critical process of building and holding 
a research competence. Dr. Donald M. McArthur, representing the Office of 
the Director of Defense Research and Engineering (DDR&E) before the Sub- 
committee, stated that the quality of a laboratory M . . . really comes 

down in the end to a number of criteria, but an overriding criteria is 
mission. Does it have a sense of purpose? Does it have high-quality 
people, and are they performing well? 11 (Subcommittee, 1968; p. 151). 

With respect to the mission or purpose of Federal laboratories, 

Dr. Donald F. Hornig, director of the Office of Science and Technology, 
Executive Office of the President, spoke to the Subcommittee of some things 
that are not. He emphasized the point that these laboratories do not exist 
to "accumulate a pile of knowledge,' 1 that they do not "turn over the grains 
of sand in the Sahara Desert and examine them one by one," that theirs is 
not the purpose to do research for its own sake. Their task is to solve 
present problems and lay the foundation for the future; they must produce 
the ideas on which the next generation of their parent agencies 1 activities 
will be based. Accordingly, the laboratories need to have meaningful prob- 
lems to work on, where the end results of what they do will be visible and 
on which they could be judged. He mentioned a "multiplier effect" of the 
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product of research activities. That is, their effect is on present and 
future programs of the entire Nation usually involving expenditures many 
times greater than those directly involved in the laboratory operations. 
Because of the "multiplier effect," Dr. Hornig stated that the utility of 
Federal laboratories must be viewed in the context of their overall contri- 
bution to national progress rather than in the narrow context of the admin- 
istration of the laboratories themselves. 

The mission or purpose of any government laboratory is operation- 
ally manifested in the problems that are selected for investigation and 
solution- The speakers before the Subcommittee were in close agreement 
over the fact that problem selection was the critical and most difficult 
task facing laboratories and especially the laboratory director. They noted 
that the input for making the decisions needed to come up from "the bench" 
and the laboratory management as well as from echelons above the laboratory. 
Hornig stated that it is not usually true that a person sitting at the top 
of the pyramid like himself, could say what the problems really are and 
what real technical problems are "soluble next year". He continued: ", . . 

whether you make progress depends on picking the right detailed problems, 
those which are ripe for solving at a given time. That requires keen tech- 
nical insight." (Subcommittee, 1968; p. 9.) 

Problem selection with respect to the Navy's personnel research lab- 
oratories is an extremely complex procedure that requires interaction between 
the highest echelons of Navy management, several levels of middle management, 
and the outlook at the "bench" in the laboratories themselves. Problem 
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selection depends, first, on establishing goals and objectives that define 
the total R&D program for the laboratories. This is accomplished, in turn, 
by considering guidance from program sponsors and planning documents, spe- 
cific consumer requests, policy changes, and the results of task-force in- 
vestigations (such as the recent Blue Ribbon Defense Panel recommendations). 
On the other hand, consideration in program definition must be given to the 
current ongoing program, the timing of requirements and expected completion 
of products, the capabilities of the laboratories, their stated missions, 
and overall funding constraints. Finally, room must be provided for self- 
generated ideas. 

Once the goals and objectives of the overall program have been de- 
fined, problem selection will then depend on priority determination and 
an allocation of the necessary resources. Again, on one hand, the poten- 
tial impact, significance, and payoff of the solution of a problem are 
prime considerations, along with requests from program sponsors and con- 
sumers. The current state of the art, requirements for additional research, 
and the probability of success are also important factors. Again, there 
are, on the other hand, the capabilities, specialization, and performance 
history of those who would work on a problem and the investment required in 
facilities, capital equipment, and personnel. Finally, there is a broad 
allocation problem of the division in resource committment between research 
and development, to the extent that these can be differentiated (Gaver & 
Srinivasan, 1970). Underlying the entire area of problem selection is the 
question of how to do it rigorously. 
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Returning to the testimony before the Subcommittee, the most critical 
questions in the effective utilization of Federal laobratories , according to 
Hornig, were: 

(1) The choice of problems, their significance, and the feasi- 
bility of finding solutions through research and development; 

(2) the creation of capabilities in the laboratories which can, 
in fact, solve the most difficult problems; and (3) the trans- 
lation of the results of the laboratories r work into action in 
either the public or private sectors. (Subcommittee, 1968; p. 9.) 

Mission orientation in government laboratories usually means that 

there is a parent agency whose mission is being supported by the laboratory. 

Hornig (Subcommittee, 1968; p. 13), for example, has said: 

An effective R&D program involves a dynamic give and take be- 
tween the laboratory and its parent agency. It must not only 
carry out assigned tasks, but spell out the tasks which need 
to be performed; it must be a source of ideas for its parent 
agency and help the agency to put the laboratory f s output into 
practice. All of this requires a very close identification 
between a laboratory and its sponsoring agency. 

Mission orientation in R&D, on the other hand, may have both positive 
and negative aspects (Nichols, 1971). The positive aspects of this rela- 
tionship were explained in the Bell report (Bureau of the Budget, 1962) as 
follows : 

Federal operations, such as the governmental laboratory, enjoy 
a close and continuing relationship to the agency they serve 
which permits maximum responsiveness to the needs of that agency. 

Such operations accordingly have a natural advantage in con- 
ducting research, feasibility studies, developmental and ana- 
lytical work, user tests and evaluations which directly support 
the management functions of the agency. Furthermore, an agency- 
operated research and development installation may provide a 
useful source of technical management personnel for its sponsor. 

However, Hornig, speaking before the subcommittee, also admitted that 

almost all reports were in agreement that there is generally excessive 

administrative control and not enough freedom given to the directors of 

laboratories. He attributed this not only to civil service or other 
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rules but to past practices and procedures. He mentioned specifically 

as a difficult problem the relationship between military and civilian 

personnel in the direction of DOD laboratories, a problem which was 

accentuated by the policy of military rotation applied indiscriminately 

to technically qualified officers. 

The difficulty in handling parent agency-laboratory relationships 

in a beneficial way and the persistence of problems in this area were 

commented upon by Glass (1970, p. 25) in reference to the findings and 

recommendations of the Blue Ribbon Defense Panel issued on 1 July 1970. 

After almost nine years of a concerted effort to improve 
the climate and performance of in-house laboratories, the 
findings of the Blue Ribbon Defense Panel are somewhat 
depressing. Some people believe that, as long as in-house 
laboratories are organizationally imbedded deep within 
the Military Departments, efforts to achieve long-term 
improvements in laboratory operations are sure to fail. 

They feel that there is no way to protect the laboratories 
from the staggering overload of bureaucratic red tape 
and from diffuse, fragmented middle management levels 
that are apparently unable to delegate needed authority. 

On the other hand, many people believe that the close 
interaction of laboratories with their sponsors and 
their Department's needs would be unnecessarily perturbed 
if new organizational barriers were created. 

The preceding discussion on the mission orientation of Federal 
laboratories and their parent agency relationships have involved the area 
of laboratory controls. That is, the mission orientation necessitates 
selectivity in problems to be investigated, while negative aspects of 
the parent agency-laboratory relationships are reminiscent of the dys- 
functional aspects of pyramidal organizations described by Argyris . Are 
there characteristics of Federal laboratories that are also similar to 
the financial responsibility pressure mentioned by Hall and Lawler? 
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One possible means of exerting pressure of this nature was vol- 
unteered by Dr. William B. McLean, Technical Director, Navy Undersea 
Warfare Center, to the Subcommittee (1968). He felt that competition 
among laboratories in given areas was necessary so that the effectiveness 
of the laboratories could be demonstrated on a comparative basis. The 
process, he thought, would provide high incentives, high motivation, and 
the "only nonsubj ective measure of effectiveness in R&D . . . . " (Sub- 
committee, 1968; p. 80). Daddario, however, suggested that it would be 
difficult to implement such a plan of duplicate or overlapping laboratory 
facilities so that Congress could effectively deal with it in the final 
determination and allocation of funds. McLean suggested that the situa- 
tion actually existed in some areas — for example, the Air Force probably 
provided much competition to NASA. 

McArthur, commenting on McLean's suggestion, took a different 
approach. He remarked that competition for funds exists within the DOD 
laboratory structure and that this runs the gamut from no competition 
for funds to laboratories that are almost totally competitive for funds. 
Noting that the latter tends towards laboratories becomming "job shops" 
with few or no longer range programs, he added that the optimum is some- 
where between the two. 

Mr. E. 11. Glass, DoD Assistant Director for Laboratories, who 
was accompanying McArthur, commented that a laboratory should have to 
compete for at least 25 percent of its funds to stay healthy. The 80 to 
90 percent of their funding that some Navy laboratories were competing 
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for from various customers (at that time) was thought to be excessive. 

He observed also that the situation with the Air Force laboratories, 
which received M block funding" and were not dependent at all on customers, 
could be made healthier if they had to compete for some of their funds. 
Competing for funds from a customer, he explained, resulted in a healthier 
situation by coupling laboratories more closely with their customers. 

Thus, it is obvious that pressures not unlike "contract pressures" 
experienced in the non-Government laboratories also exist in the Federal 
laboratory structure. Accordingly, the organizational committment of 
scientists should be a characteristic of Federal as well as civilian 
laboratories. That this is a characteristic of effective Federal labora- 
tories has already been mentioned in some of the comments reported above. 

The general environment of Federal laboratories, when it has been 
bad, has been due to weakness in the areas of pay and professional benefits, 
lack of challenging assignments and personnel recognition, and the arbi- 
trary use of bureaucratic regulations with resulting frustrations and 
inhibiting effects (Glass, 1970). To this list might be added inadequate 
on-the-job support, as reported by the standing committee to the Federal 
Council on Laboratories. The standing committee also stated that such 
things as job titles, the retention of rights to patents, consulting fees 
and honoraria, security matters, and freedom to consult, lecture, and 
teach were considered relatively unimportant by the vast majority of 
1,025 Federal scientists who were respondents to a questionnaire. (Sub- 
committee, 1968). 
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Characteristics of a desirable organizational environment that 

would maximize effectiveness were prepared by a Bureau of the Budget 

committee in conjunction with the Bell report (Bureau of the Budget , 1962) . 

The basic requirements for such an organizational climate were (Glass, 

1970, p. 33 and Appendix IV): 

Reputation and atmosphere 

Excellence of staff and its direction 

Importance of facilities and support 

Clarity, challenge and urgency of objectives 

Maintenance of professional caliber of operations 

Professional leadership by professionals 

Professional recognition and encouragement 

Encouragement and support of freedom of inquiry and method 

Advancement on merit 

Absence of unproductive regulations and reports 



C. DIMENSIONS OF THE SCIENTIST-LABORATORY INTERFACE 



The preceding analysis dealt with laboratories in general and 
more specifically with Federal laboratories. Some factors tending to 
enhance or inhibit the effectiveness of these laboratories were identified 
and discussed. When R&D laboratories are considered in the aggregate, it 
may be possible and even advantageous to think of individual laboratories 
as basic entitites with characteristics and "personalities" of their own. 
From the standpoint of assessing the relative effectiveness of laboratories, 
this method may also be appropriate. In fact, a DoD program is in progress 
that takes this approach (Glass, 1969). Individual laboratories are being 
ranked on the basis of their effectiveness or excellence and the relation- 
ships of various laboratory characteristics to this measure of relative 
excellence are being examined. On the other hand, when one 1 s concern is 
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in assessing the absolute effectiveness of a specific laboratory and also 
involves administrative and managerial approaches for upgrading that 
laboratory ! s capabilities, he must look for different items of information. 
He must look below the overall laboratory level to that area of dynamic 
interaction between the characteristics of the laboratory and the char- 
acteristics of its individual professionals who man the laboratory. This 
level might be called the level of the scientist-laboratory interface. 

The characteristics of a laboratory might include such factors as 
its geographic location, whether it is located separately or within a 
larger military complex, overall size, budget and source of funds, sub- 
ject area of research, location in the organizational chain of command, 
position on the research or developmental end of the R&D continuum, bal- 
ance between in-house and contracted-out research, type and degree of 
external control exerted on the laboratory, and so on. These variables 
may be important in the comparative evaluation of laboratories. At least, 
they provide convenient and reasonably reliable categories for describing 
individual laboratories. But when one is dealing with one laboratory, 
many of these variables become constants which are of interest only to 
the extent that they define the boundary conditions within which the 
laboratory must operate. 

On the other side of the interface, a great deal of research has 
been conducted regarding the characteristics, attitudes, and motivations 
of individual professionals in R&D. In the Hall and Lawler (1971) studies, 
for example, satisfaction of professionals surveyed was measured on a 
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hierarchy of human needs. Their questionnaire yielded scores for security, 
social esteem, autonomy, self-fulfillment, and a composite score which 
was a single average of the part scores. Job involvement was measured by 
an attitude scale. As in the Hall and Lawler studies, many other attempts 
have been made to find behavioral or output correlates of such personal 
attributes. These output criteria might be mobility, promotion, or some 
measure of volume, quality, and influence of an individual's work. But 
here again, in the case of a specific laboratory, one is not sure whether 
the previously found relationships will hold between the personal variables 
and the criterion variables. Why? First, effect of personal variables, such 
as satisfaction and job involvement, on output are dependent on other 
characteristics of the individual. For example, the relationship of satis- 
faction to some measure of output in laboratories may depend on the age 
distribution of the laboratory's scientists. Second, the relationship of 
the personal variables to the outcome variables is mediated by variables 
that are dimensions of the work or laboratory environment. And third, 
since much of this research is correlational, it is often difficult or 
impossible to infer the causal direction of the relationships established. 

For example, how much of job involvement is something intrinsic to the 
individual that he brings to the job and how much the result — the output — 
of his experience with the laboratory work environment? 

Accordingly, when interest is centered on a specific laboratory, 
the crucial relationships are at the level of the scientist-laboratory 
interface. Detailed information is necessary which states how manipulation 
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of the alternatives available to laboratory management affect different 
scientists to bring about specific outcomes or results. It is especially 
important to discriminate between inputs and outputs. The laboratory’s 
mission and tasks, externally imposed, are inputs which can be considered 
as constants, as previously explained. The scientist and what he brings 
to the laboratory in his person is the other input category. The labora- 
tory environment or climate in which the scientist and the laboratory’s 
organizational structures, procedures, and facilities interact might be 
called process. The result of the interaction is output, which includes 
the tangible products and services of the laboratory and also such secon- 
dary outcomes as experienced manpower and changes in the attitudes and 
outlook of the scientists. Thus, there is a mechanism for change, since 
the input (the scientist) will have been changed in the process for the 
next iteration. Similarly, the outcomes will provide feedback to the 
laboratory management for suggested changes in the operational environment. 
These categories--input , process, output, and feedback — might be called 
stages. They are schematically diagrammed in figure 1. 




Feedback 



Figure 1. Simplified structure and dynamics of laboratory operations. 
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The stages can be further broken down into dimensions and variables 
as shown in figure 2. Of the input variables, education can be broken 
down into PhD, non-PhD, and Engineer categories. Pelz and Andrews (1966) 
used these categories and further classified their subjects into (1) PhD f s 
in development-oriented laboratories, (2) PhD f s in research-oriented 
laboratories, (3) non-PhD 1 s in development-oriented laboratories not 
dominated by PhD's (called "engineers’'), (4) non-PhD's in PhD“dominated 
laboratories (called "assistant scientists") , and (5) non-PhD scientists 
in research-oriented laboratories not dominated by PhD's (all in Govern- 
ment settings). Age is a continuous, quantitative variable which can be 
categorized, if desired. Orientation is an individual's preference for 
research or development activities. Specialization can be identified by 
an individual's area of specialization and further subdivided into the 
number of subareas in which he feels competent to do work. Professional 
style refers to the type of problems and the approach one prefers. The 
other dimensions — attitude, ability, personality, physical makeup — are 
named to fill out the list. Numerous other variables might be listed. Of 
the listed dimensions, education and age are parameters that can be most 
reliably obtained for personnel in a laboratory. Orientation, speciali- 
zation, and professional style could be readily obtained by some form of 
questioning the technical staff. The others are more complex, more diffi- 
cult to assess, and more unlikely to be related to the process and output 
variables in any simple, predictable manner. 

The process variables are those that are characteristic of the 
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Figure 2. Breakdown of laboratory operations into dimensions and component variables within stages. 
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procedures and working environment of the laboratory. These are the 
elements that can be manipulated to change the output variables. Orien- 
tation refers to the research vs. development bent of the laboratory. 

The proportion of short vs. long-range programs could be another variable 
in this dimension. The important dimension of control can be identified 
by a number of variables such as loose vs. tight, number of checkoffs 
required for decision making, degree of influence assertable by the indi- 
vidual scientist in decision making, and sources of ideas and persons 
involved in -goal setting. The time span of discretion might be another 
variable in the control area. Work unit characteristics can be defined 
by the degree of homogeneity or similarity of the individuals within the 
unit and the size and age of the unit. Individual assignments can be 
classified into the proportion of time spent in R&D work vs. adminis- 
trative tasks, the proportion of technical time spent in research vs. 
development activities, and the number of R&D functions to which one is 
assigned. The communication structure and dynamics of the laboratory 
can be defined in terms of the number of contacts an individual has within 
his own work unit, the number of contacts he has outside the work unit, 
and the frequency of these contacts. Rewards can be looked at from the 
traditional aspects of promotion, pay, and fringe benefits; they might 
be identified by some less tangible forms of reinforcement, such as re- 
cognition. 

The problem in using the process variables in any analysis is 
their method of measurement. On one hand, many values could be determined 
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by asking the management and by going through organization charts and 
other laboratory documentation. But research, and the practical ex- 
perience of many persons, has shown that official statements and the 
opinions of the individual workers in the organization may vary con- 
siderably. Since the framework of analysis described above involves the 
interaction between the individual scientist and the laboratory operating 
environment, it would probably be necessary to ask the scientist to scale 
most of these variables from his own point of view. 

Turning to the output variables, those listed under product or 
service require further operational definition before they can be mean- 
ingfully evaluated. This problem will be taken up in the next section 
of this paper. The attitude change variables, again, would have to be 
assessed on the basis of solicited individual opinions. 

The foregoing list of variables borrows heavily from the monu- 
mental Pelz and Andrews (1966) work on scientists in organizations; their 
categorization by dimensions and stages and the inferred direction of 
causation do not. The description and discussion of general trends in 
the R&D field, and especially in laboratories, plus the detailed break- 
down of the structure and dynamics within a laboratory, should provide 
the necessary background and framework for analyzing the specific prob- 
lems posed in this study and making appropriate recommendations for their 



solution. 
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III. EVALUATION OF LABORATORIES 
A. PURPOSE, NEED, AND APPROACHES 



Evaluation of the effectiveness of laboratory performance is a 
responsibility of those who manage the laboratories and the parent agencies 
that support them. While the need — or requirement — and even the desire 
to do so may be strong, the question of how to conduct the evaluation is 
one that has not found a universally acceptable answer. Even the question 
of M why n may not find a convergence of answers. Glass (1969, p. 1) writes: 

R&D managers have been trying for many years to devise 
better yardsticks to measure the effectiveness and utility 
of laboratories. Most managers agree that such techniques 
are needed, but few can agree on how such appraisals can or 
should be made. 

The evaluation of laboratories was a question of prime importance 
to the Subcommittee on Science, Research, and Development as an aspect of 
fostering the best utilization of Federal laboratories. Most of the wit- 
nesses before the Subcommittee were asked to comment upon their philosophy 
and approaches to laboratory evaluation. In addition, the witnesses were 
asked to submit answers to written questions provided them following their 
appearance before the Subcommittee. One of the questions dealt with the 
problem of evaluation. The views expressed in their statements provide 
excellent insight into the mechanisms of evaluation and the philosophy be- 
hind the approaches taken. 

Hornig (Subcommittee, 1968; p. 22), provided a broad overview of 
the appraisal of laboratories: 

Appraisal of in-house laboratory performance is 
a normal responsibility of agency management. . . . 
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Procedures in use in the Government . . . involve such 
techniques as visits to the laboratory by teams of agency 
management representatives; evaluation of results by 
agency management and — especially where more basic re- 
search is involved — by outside advisory groups; and 
continuing reviews of laboratory operations through re- 
ports, audits, conferences, day-to-day contacts, and 
so on. In some cases, development activities lend them- 
selves to controlled scheduling procedures such as PERT, 
but such control methods are generally not applicable to 
research near the basic end of the spectrum. Evaluations 
are necessarily qualitative rather than quantitative to 
a considerable degree, and involve judgments based on 
such factors as experience and comparison with good 
practice elsewhere. 

The prime objective is not the application of any 
specific set of administrative techniques, but the eleva- 
tion of the quality and efficiency of administration of 
Federal laboratories totally, including such matters as 
maintenance of challenging and relevant laboratory missions, 
elevation of salary scales to attract first class managers, 
and securing sufficient freedom for laboratory managers. 

McLean (Subcommittee, 1968; p. 84) brought out the difficulty of 

finding a proper criterion to measure in-house laboratory effectiveness 

that could be compared with the profit-loss ledger of the profit-oriented 

civilian enterprises. He also mentioned the need for subjective appraisals 

and review committees, as follows: 

For any organization or individual to feel success- 
ful there must be some mechanism for measuring the degree 
in which they have fulfilled their goals. In an organiza- 
tion which is profit-oriented, such an evaluation is straight- 
forward, rigorous, and simple. If the figures are in the 
black, all associated with the organization are happy. If 
they are in the red, or tending toward the red, then some- 
thing must be done to rectify the situation. Government 
organizations, military organizations, educational institu- 
tions, and research and development activities, whenever 
they are adequately removed from the profit-making pressures, 
have a more difficult time in establishing a proper evaluation 
of the effectiveness of their processes and results. For 
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all such organizations I believe the evaluation must 
be on the basis of competition similar to that in- 
volved in making a profit. The fact, however, that 
results cannot easily be expressed in terms of a 
single variable, such as money, tends to make the 
evaluation process much more difficult. Governments 
are judged by history, and military organizations by 
wars. These are very harsh and final judgments and 
do not provide a very adequate, self-rectifying mech- 
anism. 



In essence, the appraisal of contractor and 
laboratory performance is limited by the capabilities 
of the individuals available to perform the appraisal. 

Of necessity, an appraiser must be a person who has 
been very successful in the field being evaluated. 

Yet, every appraiser has his own set of biases and 
believes that his own approach is the only correct 
approach. The competitive system is the only appraisal 
system that leaves open the possibility of innovation. 

In spite of these difficulties in evaluation, the 
Navy is setting up technically competent review commit- 
tees to review laboratory performance. The effect of 
these committees on laboratory performance has yet to be 
evaluated . 

In his testimony before the Subcommittee, Astin acknowledged the 
need to have some mechanism for '’rating” the many laboratories within his 
organization, the National Bureau of Standards, and implied that the pur- 
pose of such ratings was to do something about strengthening ineffective 
management and leadership and to encourage those that were strong. In 
his written comments, however, he presented (Subcommittee, 1968; p 69) an 
excellent discussion of the mechanism of program review as an evaluative 
process : 



. . . An annual or sometimes more frequent series of 
reviews of all of the significant programs of the Bureau 
are made by operating personnel to top NBS officials. 
Generally programs are closely related to one or several 
organizational units. The program of each division is 
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also subject to review at least annually by Advisory Com- 
mittees of the National Academy of Sciences. Finally, an 
outside evaluation of the Bureau as a whole is made by a 
Statutory Visiting Committee which reports annually to the 
Secretary of Commerce on the efficiency of NBS operations. 

At the program reviews those responsible for the 
program discuss their past accomplishments, present work 
and future program plans. Information is given and ques- 
tions raised concerning the adequacy and competence of the 
staff and future requirements; similar matters are covered 
in respect to facilities, equipment, funds and other re- 
sources. Program requirements and priorities are discussed 
in respect to their relationship to national needs. On the 
basis of all of the above, the present health and future 
outlook of Bureau programs are assessed by NBS officials, 
priorities are set, and appropriate allocations and commit- 
ments are made. Occasionally decisions are made that some 
programs of substantially diminished importance have out- 
lived their usefulness. When such decisions are made, the 
staff members involved are reassigned when possible or re- 
duced in force when reassignment is impractical. 

Other speakers before the Subcommittee also referred to project 
and program reviews and evaluation panels and advisory groups as mechanisms 
for conducting the assessment of laboratory effectiveness. Some, particu- 
larly the NASA representative, emphasized continuous and close technical/ 
professional communications between the headquarters and the laboratory as 
a means of appraising laboratory performance. The comments of MacArthur 
(Subcommittee, 1968; p. 151) were of particular relevance to this study: 

When we evaluate the quality of a laboratory there 
are many criteria we use. The three services, the Air 
Force, the Army, and the Navy, have advisory groups which 
periodically review the programs of the laboratories and 
come up with recommendations in terms of whether they are 
above standards, below standard, fine, mediocre, whatever 
they might be. 

We also look at whether they have meaningful missions. 

Secondly, within the Department of Defense at the 
D.D.R. & E. level where I come from, we look at programs 
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from a programmatic standpoint, from a technical stand- 
point, and at that point we look at the contributions 
the laboratories are making to that program. 

Thirdly, as you no doubt know, some of our labor- 
atories are involved at only one end of the R. & D . 
spectrum, research and technology. Other laboratories 
are involved throughout the whole R.&D. spectrum through 
engineering development and test and evaluation. 

Now, one of the criteria we use when a laboratory 
is involved at the research and technology end of the 
spectrum is how much of their output over the last few 
years has been incorporated in some of our systems devel- 
opment programs. 

In a laboratory that is involved in engineering 
development, we look at the effectiveness of the systems 
or hardware they have developed or managed, and, lastly, 
we look at the individual laboratory director’s inde- 
pendent research program and look at how he has managed 
his funds, what he has done, where he has invested them, 
but the real test is how much business he got based on 
those investments he has made. 



Among the supplemental questions addressed to MacArthur, was one 
that asked him to explain the DoD system for evaluating contractors that 
provide R&D and to state to what extent the system should or could be used 
in appraising the work of the in-house laboratories. MacArthur, in his 
reply, described the DoD Contractor Performance Evaluation (CPE) Program, 
a system designed to provide an orderly and uniform technique of determin- 
ing and recording the effectiveness of contractors primarily for hardware 
development and production. In evaluating the appropriateness of the CPE 
procedure for in-house laboratory appraisals, MacArthur ’s comments (Sub- 
committee, 1968; p. 180) were particularly illuminating: 

For those contracts where the end product is new 
technology or new scientific findings, CPE can be utilized, 
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if the Military Departments consider it desirable. How- 
ever, a less formal evaluation of such contracts is gen- 
erally utilized. This is usually in the form of a sub- 
jective appraisal by the project monitor. 

The CPE System appears to be most suitable for 
evaluating specific programs for which there are meaning- 
ful performance standards and mileposts. I might add that 
similar criteria are utilized to evaluate the hardware 
development laboratories, although in a different form. 

In other words, the CPE System is designed to evaluate 
performance on a specific well-defined project or program. 
Its purpose is not to evaluate over-all organizational 
effectiveness . 

Much of the work of Defense laboratories is in the 
areas of long-range research and technology. Thus, a 
system such as CPE would not be generally applicable. 

One advantage we see in the establishment of military prob- 
lem oriented weapon centers is that the utility of their 
output can be measured fairly directly on a real-time 
basis. Practically everyone knows and can measure the tre- 
mendous productivity of a NOTS , China Lake. Its output 
goes directly into the military inventory. On the other 
hand, the output or product of a research laboratory is 
much more difficult to assess. Many years may pass before 
the utilization of new science or technology can be meas- 
ured meaningfully. 

The principal method used for Defense laboratories 
is peer rating or evaluation, either by in-house people, 
those on the outside, or combinations of both. This is 
only part of the story, however. Program evaluation in 
terms of need, priority, technical content, and similar 
factors probably have a greater bearing upon the appraisal 
of laboratories than direct institutional evaluation. 

Through program evaluation, one usually makes decisions 
on resource allocation which ultimately determine the future 
of that laboratory responsible for the program execution. 



The comments to the Subcommittee extracted above were representative 

of the others who testified. The needs and purposes of the evaluations 

seemed to fall under the following categories: 

Determine the quality and efficiency of the administration 
Evaluate the meaningfulness of programs 
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Determine the health and future outlook of programs 

Look for challenging, relevant missions; adequate 
pay of professional staff; and freedom for the 
laboratory manager 

Set priorities and determine allocation and commit- 
ment of funds 

Look for laboratories and/or programs that have 
served their usefulness to shut them down 

Shore up the management and leadership where necessary; 
give encouragement to good leadership 

Provide guidance to the laboratories 

Determine adequacy and competence of staff and future 
requirements 

Determine adequacy of facilities, equipment, funds, 
and other resources. 

They all tended to agree that evaluation of the research and tech- 
nology end of the R&D spectrum is the difficult problem, that this has to 
be done subjectively/qualitatively, and that some sort of panel of experts 
is an appropriate mechanism. There was an underlying current that program 
evaluation and institutional evaluation are different but that program re- 
views over a period of time can serve the function of institutional reviews. 
Perhaps the distinction between program reviews and institutional reviews 
lies in the criteria available for evaluation. Program reviews have more 
specific goals against which progress towards those goals can be evaluated. 
Institutional reviews, however, emphasize such intangibles as overall quality, 
relevance of work attempted, and technical competence. If this were so, 
then the distinction between program reviews and institutional reviews might 
be one of performance vs. quality. It should be noted that good performance 
on the part of a laboratory does not necessarily ensure effectiveness or 
quality in the product, even in hardware systems. (This matter will be 
further elaborated under the discussion of evaluation criteria.) 
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B. ADVISORY PANELS 

Since the use of advisory panels seems to be generally accepted as 
a method for evaluating laboratories, a more detailed examination of the 
process would seem to be in order. While the panels are usually termed 
advisory, the term may mean anything from the simple lack of directive 
authority, at one end, to advisory in the literal sense of the word at the 
other. Usually, the distinction is closely associated with the level of 
the convening authority of the panel, tending to be more closely allied to 
direction monitoring, and decision-making processes when it is formed by, 
and reports to, a headquarters above the laboratory. Another dimension 
underlying the panel system is the constitution of the panel — that is, the 
source and types of persons making up the panel. And finally, closely re- 
lated to these two dimensions, is that of the basic purpose of a panel. 

The following interchange between Daddario and MacArthur at the 
Subcommittee's (1968, p. 152) hearings highlights the use of advisory 
panels at the highest levels in the services: 



Mr. Daddario. You talk about judgment by peers. Who 
are they and how do you get them together? 

Dr. MacArthur. There are three — let me address each 
service. In the Air Force we have a board of advisers which 
is a panel of the scientific advisory board and they are 
from the outside. They involve individuals from the indus- 
trial world, individuals from the universities, and from 
nonprofit organizations. 

In the Navy we have the technical evaluation board 
which is a part of the Naval Research Advisory Council. 

Again, all members of this advisory board or technical 
evaluation group are from outside the Department of Defense. 

The Army is different in that they have a group which 
is composed solely of all in-house individuals and they per- 
form an appraisal every 3 years. We call it the triannual 
survey group. 
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These are the advisory boards I referred to. 

Mr. Daddario. How are the laboratory directors in- 
volved in the evaluation, if at all? 

Dr. MacArthur. Well, these advisory boards, they do 
more than read papers. They actually visit the labs. They 
talk to the laboratory directors and the key people at the 
laboratories to see what they are involved in, what they 
are doing, what their mission is, what they have contributed 
in the last year, and what they intend to work on in the 
following years, and why. 

One of the biggest problems we have been having in the 
past years was that some of these laboratories weren’t in- 
volved in important questions. That is one of the things 
we have been stressing, that they have to get more involved 
in important military problems. 



Here, while there is obviously interaction between the panels and 
the laboratories, the purpose of the panels seems generally to be to advise 
the responsible command above the laboratory level and to encourage the 
implementation of the command’s policies — e.g., become more involved in rel- 
evant military problems. The evaluative nature of the panel is especially 
evident in the Army’s case by its constitution. On the other hand, the 
following exchange between Daddario and Astin (Subcommittee, 1968; p. 50) 
very definitely implies a panel that is responsive at the laboratory direc- 
tor’s level and serves at his discretion. 



Mr. Daddario. How do you bring the individuals to- 
gether and how do you allow the people who are involved in 
the Federal laboratories to participate? Yesterday we had 
some testimony on the need to do this more often than we 
do, not only because you can get different points of view, 
but also because laboratory directors would find out more 
what was going on and they could improve their own manage- 
ment activities. 

Dr. Astin. Well, it is my feeling that every labora- 
tory manager or director should have some responsibility 
for formulating at least a portion of his program. In gen- 
eral the laboratory supports the mission of the agency, and 
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to be effective, it has to be responsive. But if the 
program is to be dynamic and effective, then formulation 
of some portion of this program has to be under the con- 
trol of the laboratory manager. 

At the same time it is desirable, I think, for him 
to have mechanisms for seeking advice from experts in the 
technologies involved. In general advisory committees to 
laboratory managers have proved very helpful. 



The discussion in section A, above, also emphasized the role of 
panels in program formulation and evaluation. One of the "purest" examples 
of the use of panels for such purposes is the practice in some Federal 
agencies to assemble panels of specialists to evaluate proposals for Federal 
support. The problem that may arise when scientists themselves are used 
as the panel members is explained by Nichols (1971, p. 33): 



Some representatives and senators have been asking 
searching questions about what they see as an overelaborate 
"buddy system" masquerading as a scientific "judicial system". 
All too few people in Congress understand the enormous, largely 
unpaid efforts invested by scientists in the panel review sys- 
tem used by most agencies to evaluate proposals for federal 
support. Worse still, the dedication of most of these review- 
ers to high quality is just not grasped by a number of influ- 
ential members and staff of congressional committees. Log- 
rolling, it seems to some congressmen, is being increasingly 
substituted for what were once rigorous standards of scientific 
quality and productivity. 

********* 

Despite the valiant and often insightful efforts of a 
few scientist-analysts, Congress sees little progress being 
made in solving the difficult "apples and oranges" problem of 
allocating resources to and within various federal R&D areas. 



Nichols 1 comments suggest that advisory panels play an important 
decision-making role in the Federal R&D effort. When this is the purpose 
of a panel, Gideonse (1970) adds another dimension to the use of panels 
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for program evaluation. He maintains that the scientist-panel approach 
to decision making is only proper for ncm-mission-oriented agencies and 
those supporting fundamental research. But when agencies have more spe- 
cific missions, persons selected to serve in an advisory capacity must 
have the expertise required to make the decisions involved, especially of 
a policy setting nature. 

Thus, the use of advisory panels in the management of R&D activities 
is a complex procedure involving at least a consideration of the purpose 
of the panel, the agency that it supports, and who shall constitute the 
panel. Finally, as mentioned in section II A, top-level committees formed 
as "overseers" of research may well be a symptom of deterioration in the 
R&D process and not a cure (Argyris, 1968). 

C. CRITERIA 

Criteria to be used for evaluating laboratories are closely linked 
to the purpose of the evaluation. In addition, criteria are of little 
worth if there is no way to provide measurements which indicate a position 
on the criterion for a given laboratory. Thus, such terms as measures of 
effectiveness, figures of merit, cost/benefit ratios, and so forth arise 
to fill this need. Unfortunately, the possibility of finding appropriate 
measurements — "yardsticks , " according to Glass — become more remote as the 
purpose of the evaluation becomes global and abstract. In fact, it becomes 
difficult just to find appropriate criteria, much less methods for measure- 
ment, when such is the case. Take, for example, the desire to determine 
the technical excellence of a laboratory. What criteria does one use to 
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to do this? And what measurements are to be taken to denote the position 
of a laboratory on the criterion? 

One approach is to look for intermediate, more readily quantifiable 
criteria as estimates of the more general, abstract criteria. This approach 
assumes, of course, that the relationships of the intermediate or part 
criteria to each other and to the higher level criteria are known. Usually 
this is not the case. This approach, when carried to the extreme, results 
in measurements of manner of performance — or simply, performance — rather 
than measures of quality (Hatry, 1970). Typically, such measures involve 
quantity of output, rate of production/output, costs, manpower utilization, 
adherence to schedules, adherence to standards and specifications, and so 
forth. As indicators of workload and managerial efficiency the measures 
may be appropriate. A dramatic example of the meaninglessness of such 
measures involves the gross national product (GNP) , presumably an indicator 
of the quality of life in the nation. A family makes a contribution to the 
GNP when its mother dies: the GNP goes up because more expenditures are 

necessary to procure the services which the mother formerly provided for 
nothing (Clausen, 1971). A similar, although not so dramatic an example, 
can arise w T hen program reviews are used for the evaluation of a laboratory 
and progress on a PERT or GANTT chart is used as the criterion of the quality 
or utility of the work being performed. In fact, the opposite is more 
likely to be true: quality is sacrificed to meet the milestones. 

Another problem in evaluating R&D work in personnel research is the 
fact that it deals with human beings and their value systems (Gideonse, 

1971). Consequently, the findings, results, and recommendations of research — 
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even the topics chosen to be research ed---are evaluated against social and 
political criteria of a very general nature over and above the scientific 
quality of the work. To find, as the Navy did some years ago, that vol- 
unteers from certain areas of the United States who were very young and 
high-school dropouts created so much trouble that the Navy would be better 
off not enlisting them has the seeds of a very touchy, social-political 
problem. On the other hand, if personnel and behavioral science research 
does not come to grips with socially relevant, politically loaded questions, 
it will be criticized for skirting the problems that "really count.” It 
is understood that the Taskforce on Manpower of the Defense Science Board 
is taking a very hard look at the adequacy of Defense manpower research in 
respect to the pressing, social-political problems that involve the Depart- 
ment of Defense. The problems involved in finding adequate criteria for 
research and program evaluation dealing with the ’’quality of life” are 
thoroughly explored by Clausen (1971) and Hatry (1970) . 

Another, and often unrecognized, area creating difficulty in assess- 
ing the effectiveness of personnel research and human f actors/man-machine 
R&D requires mention. This is the fact that the absence of problems in a 
completed system is a sign of good work in these disciplines. For example, 
an operator in a complex system may find information displayed so it is 
readily monitored; controls of the right size, shape, and location; and 
work-space and environment conducive to extended-duration performance in 
comfort. His training, job aids, work procedures, and workload are adequate. 
He has ready communication channels to other operators or superiors when 
necessary. Years of research on small, apparently insignificant research 
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problems have all contributed to the knowledge required to achieve this 
situation. An experienced and talented group, thoroughly conversant with 
this knowledge, is required to put this wealth of information into the 
system design. Yet current research of the same type, looking to the future, 
may be severely criticized by the unsophisticated observer as, for example, 
"analysis of variance" research on insignificant or irrelevant problems 
with little prospect of payoff. It is indeed a difficult problem to deter- 
mine the payoff or effectiveness of long-term research at or near the basic 
science end of the R&D spectrum. 

Returning, again, to the hearings of the Subcommittee on Science, 
Research, and Development, some of the criteria mentioned by those testi- 
fying were as follows: 

1) Comparison of a laboratory with "experience" 

2) Comparison with "good practice" 

3) How much of the output of research and technology has 
been incorporated into system development programs 

4) Effectiveness of the system hardware managed by the 
laboratory 

5) Amount of subsequent business received from an initial 
expenditure of independent laboratory research funds 

6) Peer ratings 

The majority of these criteria depend on judgment, partly or completely, in 
the evaluative process. The third listed has often been called "implementa- 
tion," and is largely anecdotal. A direct, simple approach to implementing 
the "implementation" criterion is highly likely to be seriously criticized 
as in the case of Project HINDSIGHT (Radnor, Rubenstein, and Tansik, 1970). 
Item 4 is obviously the ultimate criterion, but the only valid measurement 
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is performance in the operational environment. Ignoring the difficulty 
in doing this, there is an added problem of evaluating partial criteria 
for laboratories that support the system manager. For example, when a 
personnel laboratory provides the personnel subsystem for a system, there 
is no way to evaluate precisely the contribution of the personnel sub- 
system to the total system effectiveness. Item 5 is also closely related 
to implementation, and has a strong judgmental component. Accordingly, 
all but the last mentioned criterion reflect the context in which the 
other criteria are often used — namely, in advisory panel reviews. Peer 
ratings will be discussed below under methods. 

Additional criteria which have been used in appraising and evalu- 
ating R&D can also be compiled from other sources (Hall & Lawler, 1971; 
Hoshovsky, 1970; Pelz & Andrews, 1966; US Atomic Agency Manual, Appendix). 
A partial listing includes the following, which are shown irrespective of 
the distinction between performance measures vs. quality measures: 



Inventiveness in advancing technologies 
Quality and originality of ideas and proposals 
Discernment in determining when lines of inquiry become 
unprofitable 

Stature of individuals and organization within the scien- 
tific community 

Overall manpower levels in relation to work output 
Effectiveness of personnel policies in attracting and re- 
taining qualified technical staff 
Net change in R&D budget during the last year 
Number of new internally funded projects 
Percent of projects meeting schedule 
Number of projects/contracts renewed 
Percent of projects meeting cost budget 
Ratio of contracts won to contracts proposed 
Number of papers in professional journals 
Number of patents and patent applications 
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Number of technical reports 

Frequency of visits to laboratory 

Quick fixes, problem solution events 

Citations in the literature 

Requests for laboratory reports, services 

Percent of project completed to those started 

D. METHODS 

Two methods of evaluation have been mentioned in the discussion. 

One was the use of advisory panels, and the other was peer ratings. Since 
advisory panels use ad hoc procedures which depend on their purpose and 
constituency, as previously discussed, they will not be further considered 
for methodology. Accordingly, this section will concentrate on the peer 
rating methods. Before entering into an examination of peer ratings, how- 
ever, two other approaches suggested by the criteria involved should be 
mentioned . 

The first is system effectiveness as an indicator of the effective- 
ness of the laboratory which held primary management responsibilities for 
the system. It was said that personnel laboratories provide input only in 
the form of subsystems which cannot be evaluated in any precise manner for 
their contribution to total system effectiveness. This is true in the 
case of military hardware systems. On the other hand, the personnel labor- 
atories have prime responsibility for certain soft systems, such as computer 
models for personnel planning, personnel selection devices and procedures, 
training packages and computer-aided course materials. Traditional measures 
of worth could be applied in such instances — measures such as reduction in 
reaction time, workload, and costs and an increase in efficiency. But in 
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this initial examination of the laboratory evaluation problem, this area 
will not be further elaborated. Two reasons might be given for this de- 
cision. In the first place, such programs are less than a majority of the 
total laboratory program. Second, there is the complicating factor of 
false starts, seemingly interminable programs, and junked programs of 
this nature that must also be considered in measuring the effectiveness 
of the laboratory with respect to its own system development projects. 
Finally, as an initial procedure, a peer rating method could be devised 
to obtain judgmental evaluations of the value, quality, and effectiveness 
of programs in this area. 

The other area of evaluation that might be applied to the products 
of personnel research laboratories stems from the implementation criterion. 
The approach would be to determine the degree to which products of the 
laboratory have been accepted into system designs or into system design 
methodology. This is also a complicated area in respect to an effective- 
ness evaluation of the laboratories. For example, implementation of a 
product does not necessarily mean that greater effectiveness was achieved. 
The opposite could be the case. And again, the projects that are termi- 
nated prior to implementation decisions must also be considered in the 
evaluation. Implementation of a laboratory product in a particular appli- 
cation might also be less than 100 percent of the laboratory package, or 
the product might be significantly modified by the user before and during 
implementation. And finally, lack of implementation by the user agency 
does not necessarily mean that the product was ineffective, since many 
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factors are involved in the nonacceptance of innovations in the Navy 
(Mecherikoff & Mackie , 1970). Again, as with the system effectiveness 
measure, products which might be evaluated using the implementation cri- 
terion can also be assessed judgmentally using peer rating methods as . 
the initial step. 

The evaluation of laboratory effectiveness and quality by peer 
ratings in the DoD stems from the work of Pelz and Andrews (1966). Their 
objective was not an evaluation of laboratory effectiveness, however. 

Rather, they were interested in the organizational and individual, per- 
sonal variables that lead to effectiveness among scientists in R&D settings. 
But obviously, the quality of individual work in an organization is clearly 
related to institutional effectiveness. Accordingly, a method to evaluate 
the effectiveness of individuals in an organization is essentially a method 
for the evaluation of the quality of the institution itself. In fact. Hall 
and Lawler (1971) had R&D organization managers rate the quality of their 
individual scientists and then these ratings were combined into a global 
rating of laboratory performance effectiveness. They did this to circumvent 
the difficult problem of having the laboratory director rate himself by 
rating the quality of his laboratory. 

Pelz and Andrews had senior scientists and managers in R&D organi- 
zations rate their scientists on two criteria. These were: (1) Their 

contribution to general technical or scientific knowledge within their 
field (in the last 5 years) and (2) their overall usefulness in helping 
the organization carry out its responsibilities (within the last 5 years) . 
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Now the difficult problems that accompany a mass rating scheme such as 
this result from the fact that judges do not know all the ratees, do not 
know them equally well, and do not have equal confidence among themselves 
as to the fineness of the discriminations they could make (number of rat- 
ing units) among the ratees. Usual rating methods, however, cannot handle 
gaps in the data well and require a consistent scale of measurement among 
judges. As a consequence of these constrictions, considerable forcing of 
the raters is resorted to. The raters or judges are forced to rate all 
ratees using the same scale categories; sometimes judges are even required 
to place an equal number of ratees in each category (to prevent them from 
all piling up at the high end). Thus, the convenience of the investigator 
is satisfied with unknown, but perhaps considerable, loss in the validity 
of the ratings. That is, adding noise to a set of data, even if it is 
done in a systematic way, does not always clarify the signal. 

Pelz and Andrews (1966) and Andrews and Pelz (in preparation) des- 
cribe how they were able to devise a program based on an earlier paper by 
Ford in the American Mathematical Monthly. The Ford method allowed a judge 
to eliminate those ratees he did not know, create as many rating categories 
as he wished, and to place as many individuals as he desired in any one 
category (but not more than one-third of all individuals being rated) . As 
a result of the freedom permitted, the judges were able to space their sub- 
jects over the categories they had created and could evaluate approximately 
50 scientists on the two criteria in less than one hour without obvious 



strain. This flexibility of their rating procedure was undoubtedly a major 
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factor in the amount and usefulness of the data Pelz and Andrews were able 
to obtain. 

The Ford procedure is based on a win-loss matrix which shows the 
probability that each ratee will be ranked above or below every other 
ratee. Then, by an iterative process, weights are derived for each ratee 
so that, when ratee-ratee comparisons are made using the weights, the 
original matrix will be recaptured. Ordinal scaling (ranking) of the 
weights then provide the ranking of each ratee. The program is written 
in Fortran for an IBM 360/40 system and will handle up to 130 judges who 
may use up to 130 ranked categories in evaluating up to 130 objects 
(Andrews & Pelz, in preparation). 

Dr. Maurice Apstein of the Harry Diamond Laboratories used a modi- 
fied Pelz technique in which laboratories instead of individuals were 
ranked by peers (Glass, 1969). The peers, or judges, were professional 
technical people with considerable experience in R&D. The rankings were 
made to serve as the criterion variable against which to evaluate the size 
of the contract effort administered by a laboratory; that is, interest 
was in determining the effect that the size of the contract effort admin- 
istered by the laboratory had on the general overall excellence of the 
laboratory . 

The current DoD study (Glass, 1969) in this mold was initiated by 
Mr. Evan D. Anderson of the Office for Laboratory Management, ODDR&E, 
before his transfer to the Federal Commission on Procurement. The study 
is much broader in size and scope than Apstein 1 s. There were 250 judges 
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in the sample with up to 400 being planned for the ultimate sample. The 
breakdown of judges was as follows: 



(1) DoD laboratory directors 

(2) R&D managers and technical specialists within the 

Federal Government 

(3) Industrial technical specialists, consultants and 

professionals from nonprofit organizations 

(4) Scientists and engineers in academic institutions 

(5) Technical specialists in DoD program management and 

system project offices 

The larger number of judges permitted their assignment into the foregoing 
categories for analysis of rankings. Accordingly, judgments of laboratories 
could be made on rather specific performance factors in contrast to global 
rankings of "technical competence." This permits a finer analysis of the 
competence of the various laboratories and of particular interest are the 
rankings of the customer’s of the laboratories. 

As with the Pelz and Andrews study and the Apstein study, the ob- 
jective of the current DoD study is not simply a relative ranking of the 
laboratories. Here again, the rankings — now available on different ranking 
criteria — provide the basis, the dependent variable, the criterion against 
which many factors relating to R&D management will be analyzed with the 
ultimate purpose of achieving a better understanding of the R&D process in 



order to administer it better. 
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IV. INNOVATION IN RESEARCH AND DEVELOPMENT ENVIRONMENTS 

There is little in the scientific literature on innovation in 
R&D laboratories . Perhaps the reason for this is that scientists and 
technical personnel are by the very nature of their professions assumed 
to be innovative. Why do research on innovation among innovators? 
According to Argyris (1968, p. 354), 

There is, in short, almost nothing that can be said about 
redesigning and changing research and development organi- 
zations that is based upon empirical research and experi- 
mentation within these types of organizations. It is 
ironic but regrettably true that the very organizations 
charged with research and innovation have been responsible 
for generating little research and less innovation about 
their own effectiveness. 

Because of this situation, factors that might be important for innovation 
within laboratories will be derived from a consideration of implementation 
problems in reverse. That is, problems that arise in the implementation 
of R&D products in user organizations often involve innovations in tech- 
niques, procedures, and/or materiel. Innovations within a R&D laboratory, 
then, might be considered a special case of implementation in which the 
innovation is developed elsewhere and brought into the laboratory (even 
though the innovation might have been developed by the laboratory, itself)* 
The problems of implementation associated with innovative items 
can be studied from several approaches and to varying depths of detail. 
Mecherikoff and Mackie (1970) analyzed the problem of acceptance of inno- 
vations in the Navy as a problem in attitude change. A review of the ex- 
tensive literature on attitude change, a major area of personality and 
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social psychology, did not provide them with concrete answers or handy 
tools for overcoming negative attitudes toward innovations. While their 
study is couched in the technical terminology of attitude change, their 
structuring of the problem and recommendations — based on case studies as 
well as the literature review — is essentially a tract on salesmanship. 

So while their recommendations — actually guidance — regarding appropriate 
procedures seem to be fundamentally sound, they, like principles of sales- 
manship, are probably only as good as the skill of the salesman (the change 
advocate) in any particular situation. 

The factors involved in attitude change regarding an innovation 
are listed by them as involving at least the following: (1) The person 

advocating the change, (2) the communication or message that accompanies 
the effort, (3) the group factors which affect an individual's propensity 
for change, (4) individual characteristics that result in different re- 
sponses to innovative efforts, and (5) situational factors. Any one of 
these factors can take on many values, and there are complex interactions 
among them. Accordingly, any specific occurrence of an innovation problem 
may have an infinite number of alternative courses that the attitude change 
process might take. Because of the impossibility of placing probabilities 
on these alternative courses with any degree of confidence, an approach 
that attempts to break down the innovation problem into such fragmented 
detail does not have practical applications. The approach, of course, may 
have considerable heuristic value. 

Those involved in the Northwestern University Program of Research 



on the Management of Research and Development have taken a more molar 
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approach to the implementation of R&D products (Radnor, Rubenstein, & 
Tansik, 1970). Considering the implementation problem as a special case 
of the general phase-transition process in R&D, they reach essentially 
the same impasse as Mecherikoff and Mackie. They state that “Many new 
products, product lines, and technologies fail initially, and sometimes 
remain failures, because of the inability of the organization and its 
members to adapt their attitudes and behavior in a manner required by 
the new venture" (Radnor, Rubenstein, & Tansik, 1970, p. 969). They list 
the following factors that underly implementation problems in the R&D 
environment (Radnor, et. al . , 1970, p. 971): 



1. Recognition of the need for an item. 

2. Willingness of the individuals in the receiving unit 
to interrupt ongoing work to handle something new. 

3. Technical mismatch in understanding of the specifica- 
tions of the item. 

4. Mismatch in understanding of objectives of the project 
or task. 

5. Pre-existing relations of trust or confidence between 
the parties to an implementation transaction. 

6. Degree of involvement in stages of a project. 

7. Self-interest. 

8. Urgency. 

9. Perceived threat. 

10. Level of managerial support. 

11. Point in time at which a management committment is 
made to the project — i.e., the decision to set up a 
formal project mechanism. 

Their model of the phase-transition process that is implementation is 
more specific and worked out in fuller detail compared with the large 
"black box" diagrams of Mecherikoff and Mackie. Nevertheless, it is still 
a theoretical model of the process and not a practical, workable tool. 
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Following their critical review of the implementation problem, 
they list some methods that have been reportedly successful in implemen- 
tation, but warn that they have been used in projects that have failed, 
too. The methods are (Radnor, e_t. al_. , 1970, p. 989): 



1. Assuring that there is a clear and recognized need 
for the results at the time the project is undertaken. 

2. Involvement of the ultimate user of the results early 
in the process, and maintaining communication through- 
out the project. 

3. Focus of the direction or strategy for the project in 
an individual or a small group that can review and make 
decisions about changes in direction or level of effort. 

4. Having top management support and enthusiasm. 

5. Allowing or encouraging researchers to follow projects 
into application and make careers there, if they so 
desire . 



They conclude, from the wide diversity of methods used and the lack of 
consistent cause-effect relations between methods used and project success, 
that much more research is needed in the area before there will be a rea- 
sonable theory of implementation, let alone a handbook or set of algorithms 
for carrying out implementation. Carried over into the implementation of 
innovations within an R&D laboratory, an even stronger conclusion can be 
made: there are no administrative and managerial practices that are guar- 

anteed to succeed. 

What, then, could management do to bring innovations into the lab- 
oratory with at least some modicum of success? In the first place, the 
guidance that can be gained from a study and application of principles 
developed in documents, such as those cited in the discussion above, should 
not be ignored. But there would seem to be a lesson that can be obtained 
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from the two lists provided by Radnor e_t. al. that is especially appro- 
priate for R&D management. If we remember that the implementation prob- 
lem as discussed arises from the user environment, the picture may not 
be so bleak in the R&D environment. That is, implementation in reverse 
is not exactly the same as implementation in the forward direction because 
innovation in the R&D environment is the accepted way of life and the path 
to success. It is administered out of the environment by the dysfunctional 
control processes so aptly described by Argyris. Innovation in the labora- 
tory should, then, follow as a direct result of good administration of the 
laboratory that creates in its individual members a need for innovation, 
the willingness to give time and effort to it, an involvement in innovative 
projects, a feeling of a sense of urgency for the development and use of 
innovative approaches, and an opportunity to cash in on successful appli- 
cation of innovative techniques and procedures. These factors, taken from 
the Radnor, ej^. al. lists, are but a specific example of the characteristics 
of positive laboratory pressures described earlier from the works of Hall 
and Lawler: quality, time, and financial responsibility pressures and a 

deep sense of involvement in the total life framework of the laboratory. 

As shown by Merchikoff and Mackie, the pressure points where action could 
be taken to accomplish these ends are the individual participant and the 
group and situational factors. Referred to the laboratory environment, 
these are precisely the dimensions of the scientist-laboratory interface 
shown in figure 2 and described in section II c, above. The problem for 
the laboratory manager is to determine which of- the many interfaces, when 
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properly manipulated, will provide the most payoff in the direction de- 
sired. An approach that may solve this problem satisfactorily in the 
Navy's personnel research laboratories is given in section V B, recommen- 



dations for innovation. 
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V. RECOMMENDATIONS 

A. RECOMMENDATIONS FOR EFFECTIVENESS EVALUATION 

It is known that the usual management procedures for ensuring ade- 
quate performance of a laboratory, such as coordinating conferences, reports, 
person-to-person contacts, and so forth, are fully used in the management 
of the Navy T s personnel research laboratories. Further elaboration of this 
area as a means for effectiveness evaluation is not contemplated. 

From the preceding discussion involving the use of advisory panels 
to evaluate the quality and effectiveness of a laboratory, it is recommended 
that such a panel or panels be established."*" The purpose of such a panel 
should be to assess the health of current programs; the quality of personnel 
and the techniques being employed; the relevance and meaningfulness of the 
total program configuration of the laboratories; and methods being employed 
to upgrade facilities, skills, and capabilities. The panel should include 
technical personnel and representatives of consumer organizations, as in 
the case of the panel for the current DoD peer rating effort in view of the 
broad spectrum of projects worked on in the laboratories. The reporting 
level will probably be at the Bureau of Naval Personnel level, but the ac- 
tivities of the panel should emphasize interactions with personnel at the 
laboratory level to ensure a feeling of participatory management on the part 
of the laboratory people. It would seem advisable to have a subpanel or 
another panel at the laboratory director T s level that is composed strictly 
of technical personnel to provide advice and guidance to the laboratory as 
requested . 



1. While this* study was in progress, a Laboratory Advisory Board for 
Personnel Laboratories has been established under the aegis of the Naval 
Research Advisory Committee (NRAC ) . The primary purpose of the advisory 
board is to provide NRAC background information for its advice on Navy RDT&E 
matters to the Assistant Secretary of Navy (R&D) . Its specific areas of 
evaluation are similar to those discussed here. The members are all tech- 
nical personnel under the chairmanship of Dr. Joseph W. Rigney, Department 
of Psychology, University of Southern California. 
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In the case of individual laboratories, some estimate of the quality 
or effectiveness of its products — rather than the laboratory as an institu- 
tion — is desired. Currently, the formal reports of the laboratories are 
accompanied by an ’’Evaluation of Report" form which requests the reader to 
evaluate the report and mail in the evaluation form. A similar form exists 
for end-product evaluations that are not in the formal report series. The 
report evaluation asks the reviewer to rate the report as low, average, or 
high on several categories which are related to usefulness, timeliness, 
accuracy, and format factors. In addition, open-ended questions and room 
for comments are provided. The questions also emphasize an evaluation of 
the usefulness of the report and the usefulness of the laboratory’s products. 
Unfortunately, the return rate of the forms is essentially nil. A similar 
form was in use by the US Air Forces Human Resources Laboratory, with the 
same result. The latter has been discontinued. 

It is suggested that the Pelz-Andrews technique be modified and 
applied to the laboratory’s products as the rated items in lieu of individ- 
ual professionals or individual laboratories. The products could be rated 
at the program (task area) level or at the level of formal reports that are 
terminal, comprehensive reports of a project. Probably the latter would 
present a problem in selection or a problem in too many products for evalua- 
tion. The decision would have to be worked out with the personnel R&D 
management structure. It is further suggested that the panels be of two 
distinct types. A type-one panel would be composed of technical personnel 
from the user agencies who could be expected to be familiar with the products 
of the laboratories. This would provide the all-important consumer appraisal 
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of a laboratory's products. A type-two panel would be composed of only 
technical/professional personnel from the laboratory. This would provide 
the important factor of laboratory participation in the evaluation process, and 
indirectly a voice in the laboratory's R&D program. The technical panel 
could be supplemented by personnel from outside the laboratory who are fa- 
miliar with its products to provide another facet to the evaluation. All 
of the panels would not need to be implemented. The consumer panel would 
probably provide management with the most useful information with respect 
to effectiveness. The in-house, type-two panel would be the easiest to 
implement . 

The criteria for the ratings would be different for the two panels. 

The type-one panel would use two criteria which could be called (1) use- 
fulness and (2) influence. The usefulness criterion would include the 
area of evaluation portrayed on the Evaluation of Report form and would 
emphasize short-term, direct applications of the product. Included in the 
criterion would be factors such as usefulness, timeliness, completeness, 
accuracy, adequacy of data, adequacy of approach, convenience of format. 

The specific wording of the criterion would be worked out with personnel 
research, management personnel. The second criterion, dealing with the 
influence of the report, will emphasize its long-term usefulness according 
to its potential influence on concepts, planning, and decision-making in 
the user agencies. 

The criteria for the type-two panel would also number two: (1) tech- 

nical-scientific value and (2) mission appropriateness. The first criter- 
ion would be similar to that used by Pelz and Andrews regarding the work of 
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individual scientists. The product would be rated for its value in con- 
tributing to knowledge in the field and especially its value in furthering 
the work of the laboratory. The second criterion would permit the labora- 
tory personnel to express their thoughts on what the laboratory should be 
doing, considering its general mission. In other words, the panel would 
rate its programs as most appropriate or representative of what the mission 
calls for. The mission statements, as of this writing, are those in DDR&E 
Management Analysis Report, MAR 70-4 (pages 87 and 95). 

The method of analysis would involve the use of the University of 
Michigan program, available from the Survey Research Center, for the Ford 
analysis. The derived weights for the products analyzed could serve as 
the dependent variable for further analyses based on characteristics of 
the raters and of the rated items. Such analyses would provide a fuller 
understanding of what factors constitute the effective and noneffective 
products or programs of the laboratories. They would also provide input 
for managerial and administrative practices to enhance the implementation 
of innovative techniques and procedures. 

A difficult problem in carrying out the recommended ratings would 
be an appropriate definition of the "consumer" for a type I panel. Because 
of the diversification of the areas of specialization of the Navy’s per- 
sonnel research laboratories, potential customers will not be a homogenous 
group. Accordingly, rankings obtained through a type I panel will reflect 
the constitution of the panel. In addition, there may be an insufficient 
commonality of items rated to provide a reliable ranking over a majority 
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of the items. In the extreme case, each of a few judges may rate completely 
different items, for example. Probably the best way to overcome these po- 
tential problem areas would be to use many raters (as in the DoD laboratory 
study mentioned previously). But here, practical considerations and the 
limited number of potential raters may present other problems. In the final 
analysis, however, the rankings by a type I panel constitute a managerial 
tool for providing needed information, and any bias in the constitution of 
the rater sample should reflect the desires and purposes of management in 
conducting the ratings. 

The procedure of evaluating laboratory products by peer ratings 
could be done at periodic intervals, say approximately three years, to main- 
tain a feel for the quality and effectiveness of the laboratory's work. 

As long-range programs, it is recommended that studies be initiated 
to determine how to assess the contribution of personnel research laboratory 
products to total system effectiveness and how to use implementation as a 
criterion to evaluate effectiveness of the laboratory's output. 
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B. RECOMMENDATIONS FOR INNOVATION IMPLEMENTATION 

Interviews at the Navy's personnel research laboratories indicate 
that many specific attempts to upgrade the professional excellence of the 
laboratory are being made. They include the procurement of outside speakers, 
tutorials on specific techniques, attendance at professional meetings, 
bringing in specially qualified reserve officers for their "summer tours," 
sponsoring symposia, attendance at short courses, and the other commonly 
used procedures. In one instance, for example, the developers of the auto- 
mated operational sequence diagramming technique at San Diego went to the 
Washington laboratory to put on a tutorial on the method. The effects of 
these attempts is not known but assumed to be beneficial. (The usual case.) 

It was concluded in the section on innovation (section IV D) that 
innovation in the laboratory is a problem in manipulating the appropriate 
variables in the scientist-laboratory interface to bring about the climate 
that would, of itself, create a motivation for innovation. Then, any 
specific innovative attempt or project would have a greater chance of succeed- 
ing. The problem lies in deciding which set of variables to manipulate. 

In order to determine what set of variables to manipulate, one 
must know the important interactions and their outcomes of the input and 
process variables shown in figure 2. A comprehensive study showing impor- 
tant relationships of this type is the Pelz-Andrews monograph previously 
referred to on many occasions. A simple way to answer the question of what 
to do lies in obtaining measurements on the important variables of the Pelz- 
Andrews study for a particular laboratory and then going to the study to 
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determine what manipulations will produce the desired climate. For example, 
if one group of PhD scientists turned up in a predominantly research-or i- 
ented work unit in which the decision controls for any one person were very 
few, he would find from the Pelz-Andrews monograph that having four decision- 
controls (decreasing freedom!) resulted in a higher volume and quality of 
output for such scientists. The indicated action is obvious. 

Such an approach could be taken at several levels of complexity with 
respect to data collection. A level-one analysis (simple date collection) 
would use the following variables: 

Individual variables: Age; education; time since BA/S, MA/S, 

PhD; and area of specialization. 

Work unit variables: Identification of unit, size, predomi- 

nant orientation. 

The combination of the individual variables with the work-unit variables 
will lead to many practical management procedures for moving the climate 
toward greater effectiveness of the work unit, including implementation of 
innovative methods. The value for these variables could be obtained from 
individual records, organizational charts, and unit charters. In addition, 
management could provide estimates as to some of the control variables 
that might apply to the work units, thus enrichening the analysis. If the 
laboratory products evaluated in the peer ratings can be traced by source 
to the work units, the relationship of work-unit characteristics to quality 
of output could be analyzed using a multivariate procedure described below. 

A more complex level of analysis would involve the creation of 
appropriate questionnaires to obtain data from individual scientists in the 
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laboratories. Of particular interest would be the obtaining of additional 
dependent measures for multivariate analyses. These would be the number 
of papers published and presentations made in a specified period of time 
and the number of laboratory reports authored/coauthored in that period. 
Again, it would be desirable to identify the work unit and the individual 
with specific products or program areas that have been peer rated. Multi- 
variate analyses of the input and process variables using the peer ratings 
and individual production as the dependent variables would provide a rich 
picture of the factors within the laboratory associated with effective and 
noneffective individuals and work units. A multivariate technique, Multiple 
Classification Analysis, has been obtained from the Survey Research Center 
that is particularly adaptable to this analysis (Andrews, Morgan & Sonquist, 
1969) . This technique and the computer program listed were used for the 
analyses in the Pelz and Andrews study of scientists in organizations. 
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